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Abstract 

Large organisations, such as universities or financial institutions, generally have access to substantial 
volumes of data from various ‘transactional’ sources, such as customer service records, email, or student 
management systems. There is an ever-increasing demand on these large organisations to make rapid, 
relevant and efficient decisions. Given this growing demand, there are at least three key principles that aid 
decision-making in such large organisations: (a) the need for decisions to be supported by facts rather than 
perceptions, (b) the general expectation for organisations to be accountable and transparent in their 
decision-making and (c) the access to data and conduct of time-series analyses to enable an organisation to 
be competitive. 

The aim of this article is to give an insight into how a large research-intensive Australian tertiary 
institution has approached the collection of data to allow effective and efficient institutional decision¬ 
making about improvements to teaching and learning by creating a central data management system that 
brought together existing dispersed university databases and repositories. 
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Currently, there is a growing demand on organisations (whether corporate, government or 
educational) to make increasingly complex and significant decisions around investment, 
marketing and performance of staff and products on a day-to-day basis. Despite having access to 
enormous volumes of data from their various ‘transactional’ systems (such as customer service 
records, email, or student management systems), organisations generally do not have a consistent 
approach to consolidating all available data within a single database (Guan, Nunez, & Welsh, 
2002; Marsh, Pane, & Hamilton, 2006). 

Davenport and Harris (2007) outlined three factors that contribute to the need for 
decision-making to be supported by data: 

• Decisions must be based on facts rather than perceptions. 

• There is growing pressure (from the government and society in general) on organisations 
to be accountable and transparent in their decision-making processes. 

• A requirement for data-driven evidence and analyses in enabling an organisation to be 
more competitive in its field. 

Implicit in the current data-supported decision-making drive are the assumptions that data 
are important in guiding improvement, but also in holding individuals accountable. The pressure 
to make decisions supported by data has featured prominently, not only in the corporate sector 
but also in the higher education sector in many countries around the world (Davenport & Harris, 
2007; Guan et al., 2002; Nair, Pawley, & Mertova, in press). As a result of recent pressures, 
universities have re-examined their existing management systems due to increasing demands on 
improved data management. These demands come from a wide range of stakeholders involved in 
higher education. 

Guan et al. (2002) referred to three groups of stakeholders: university leaders and 
policymakers, administrators and academics, and external oversight agencies. However, there are 
also a number of other important stakeholders, including students, who demand information 
tailored to their specific needs. Universities generally have access to large amounts of data on a 
range of aspects to satisfy the wide range of their stakeholders. Nevertheless, in many cases, they 
do not have management systems in place to effectively consolidate the available data in 
meaningful ways. Further, Guan et al. (2002) highlighted two reasons for the lack of effective 
data-supported decision-making in many tertiary institutions: 

• the nature of existing IT systems (systems frequently put in place to cater for immediate 
needs, and lack of adaptability of these systems); and 

• a general lack of integration within the existing IT infrastructure. 

This resulted in data being located across different systems (frequently within a single 
institution) that were often incompatible (Linthicum, 2000). 

Marsh et al. (2006) and Guan et al. (2002) highlighted that having data available, 
however, does not necessarily result in effective decision-making. Guan et al. (2002) pointed to a 
number of desired characteristics of the data required by current institutional decision-makers, 
such as information being derived from a broad range of internal and external sources, and 
historical data to facilitate understanding trends. 
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Marsh et al. (2006) also cautioned that available data may not necessarily translate into 
effective decision-making that results in improvement. They pointed out that data availability 
needs to be weighed against the ‘trade-offs’ of time spent on collecting and analysing data, and 
also the costs. Marsh et al. (2006) suggested that one approach to making evidence-based (or 
data-driven) decision-making more effective and efficient is through a data integration technique, 
namely data warehousing. This is an approach that brings together data from various sources in 
one user-friendly presentation. Data warehousing originated in the manufacturing sector and 
industry and has increasingly been introduced by Australian higher education institutions (Marsh 
et al., 2006). 

Inmon (1996) defined data warehousing as an ‘integrated’, ‘subject-oriented’, ‘time- 
variant’, ‘non-volatile’ database that supports decision-making. This means that the data are 
integrated in the database from diverse and multiple sources. The data integration requires 
standardisation of all data elements, which may be time-consuming; however, when finalised, the 
database provides consolidated data across the whole organisation. The data are aggregated by 
topic and are never removed from the database. Data warehousing also provides an effective 
means of managing large amounts of data (Guan et al., 2002). 

A number of institutions (such as the University of Western Sydney, University of 
Technology Sydney, and Monash University) have started consolidating valuable student 
feedback with other key data (e.g., admissions data, retention measures, pass rates, or data from 
student management systems) in order to profile and track individual units and courses against 
predefined targets. Such an approach allows for a detailed analysis of performance at a high 
management level (within an institution) and then the ability to further investigate the causes of 
particular issues of concern. 

This article looks at how Monash University, Australia — a large research-intensive 
institution that is home to more than 57,000 students from around 130 countries (with 6 
Australian and 2 international campuses) — dealt with the current imperatives on data-driven 
decision-making. In particular, the article shows how the university approached the issue of 
centrally integrating existing data from multiple university databases to enable more effective 
analysis of previously dispersed data related to quality of education and thus aid improvement in 
the institution. 


The approach to storing and integrating data 

Monash University, like many other tertiary institutions, had substantial volumes of data 
stored in a wide range of databases and repositories. The university has recently recognised an 
increasing need to understand what was happening about a particular issue (such as pass rates or 
retention rates) across the whole institution. This created a demand for a system capable of 
integrating relevant data in a single central location to enable reporting at a whole institution 
level. As a result of this demand, the university set out to create a new business strategy. This 
involved a two-stage process. 

Stage one involved changing the institutional practices in gathering student feedback. 
This initial step led to the introduction of a new Survey Management System (Nair & Wayland, 
2005). The Survey Management System is an institution-wide evaluation system that integrates 
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data collected via online and paper-based surveys from across the institution and stores the data 
in a centrally located database. The need to understand and cater to the needs of a wide range of 
Monash University stakeholders, and thus the requirement for a robust evaluation system, was 
highlighted in a number of the university’s strategic documents — such as Monash Directions 
2025; Excellence and Diversity 2004-2008', and Quality at Monash: Values and Principles — 
and these eventually prompted the development of the new Survey Management System. 

This evaluation system, enabling integration of a range of existing university databases, 
has been recognised by a number of university leaders and academics as valuable for the 
purposes of course and unit management. Similar to other Australian universities, Monash 
University did not previously have a central repository of data. Locating data centrally provides 
an easily accessible rich source of information for decision-makers. The integrated database 
facilitates the production of consolidated reports to be used by unit and course leaders, heads of 
departments, deans and the university management, and it also more effectively assists them in 
making evidence-based judgments. The practice prior to the introduction of the uniform reporting 
process was that individual areas within the institution would embark on obtaining the required 
information from different sources where data were located. However, different variables were 
required to obtain information according to the data source and the needs of the area. This 
resulted in a lack of consistency, and neither comparative nor time series data were available 
easily within areas or across faculties and campuses. 

Following the success of gathering systematic data on student feedback, the Unit and 
Course Profiles were developed (Stage Two). These are an important monitoring tool within the 
Monash University Quality Cycle (Monash University, 2002). The Unit and Course Profiling 
system enables a single report to be generated for a unit or course. The system has two main 
features: it highlights issues and areas that may require further investigation and improvement 
and it identifies good practice and areas that may serve as exemplars of good practice. 

The profiles enable all university stakeholders to access the same information, thus 
allowing comparison between performance and decisions made with respect to particular units 
and courses. For example, the system allows identification of units that fall below a university set 
benchmark for poor performances. Such units are then earmarked by the respective faculties and 
departments for developmental work to improve the teaching and learning. Some of the strategies 
used involve workshops to improve certain aspects measured in unit evaluation, the review of 
unit outlines and assessments. The course and unit profiles structure has the advantage of 
identifying units by location, thus achieving a finer grade for improvements to specific classes. 

The Unit and Course Profiles implemented at the university uses the IBM® Cognos® 8 
Business Intelligence software suite that allows users to view different dimensions of data and 
metrics against predefined targets, and look at performance and trends in a single view. One 
reason for choosing this technology was the ease of use of the software to analyse and display 
data. 


The key features of the Unit and Course Profiling system are described as follows. 
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(1) Enables creation of searchable reports 

The reports comprise a set of indicators that can be viewed from the ‘top down’, 
university-wide, by faculty, then by campus and course or unit results as a whole. It further 
enables more detailed examination to identify outliers in each of the performance areas. The 
profiles are also benchmarked against internal and external standards and targets/comparisons. 

(2) Linked to the university’s Key Performance Indicators (KPIs) 

The metrics used in the system are driven by selected university key performance 
indicators, allowing for better understanding of these metrics. Targets/comparisons are developed 
for each metric against which performance can be measured. 

(3) Enables evidence-based improvement 

The profiles assist faculty and departmental managers to identify units and courses that 
are under-performing and thus raise an alert to the need for a diagnostic process in line with the 
university’s quality cycle to ensure that improvement efforts are appropriately targeted. 

(4) Enables better tracking of improvement actions and accountabilities 

The Unit and Course Profiles allow the tracking and management of accountabilities and 
actions to ensure improvements are made in a timely manner. 

(5) Integrates multiple databases in a single repository 

A number of university data sources were used to create these profiles (otherwise 
commonly referred to as Scorecards) for each unit and course at Monash University. These data 
sources included: unit evaluation data, DEEWR Load Files, 1 Graduate Careers Data-Course 
Experience Questionnaire (CEQ) 2 and Graduate Destination Survey (GDS), as well as a host of 
other enrolment and grade data from the student management system (referred to as Callista). 

Figure 1 summarises the features and processes that were involved in developing the 
profiles from concept to reality. The whole process evolved over a period of 2 years. 


1 Australian Federal Government Department of Education, Employment and Workplace Relations (DEEWR) Load 
Files are annual files containing specific university-wide information on courses and units. 

2 Course Experience Questionnaire (CEQ) and Graduate Destination Survey (GDS) is a combined survey that is sent 
to every graduating undergraduate and postgraduate by coursework student in Australia to establish their study 
experiences and also their post-graduation activities in terms of their employment or further study. 
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Figure 1: Process overview to establish Unit and Course Profiles 

Effective implementation of a quality evidence-based decision-making system involves a 
number of factors such as: 


• the support of senior management for such a system of reporting; 

• organisation-wide acceptance and coordinated use of data for decision-making; 

• development of a robust data gathering and storage mechanism; and 

• support by information technology infrastructure. 

An underlying driver of all these factors over recent years has been the realisation that 
data are not only required for improvement purposes, but are also increasingly tied to funding 
requirements (Nair, Bennett, & Wayland, 2006). In Australian higher education it is, for instance, 
the Learning and Teaching Performance Fund (LTPF) introduced in 2006. The LTPF funding 
represents the sum of approximately AUS$250 million, which is allocated to tertiary institutions 
over a 3-year period (2006-2008). Therefore, building effective systems to support evidence- 
based decision-making in Australian higher education institutions, such as Monash University, 
has recently been recognised to be a vital, as well as cost-effective, approach to securing 
significant amounts of government funding (Australian Universities Quality Agency [AUQA], 
2006). 


Concluding remarks 

Clearly there is need for universities to have systematic and timely access to data to make 
decisions and initiate improvements. To achieve this outcome, an effective data warehousing 
system is needed in institutions. The Course and Unit Profiles was introduced to Monash 
University following two years of planning and designing to achieve this outcome. This system 
of reporting has a number of benefits that include: 

• enabling the university and individual faculties and departments to get a snapshot of 
information about each unit or course per semester/year and also to identify improvement 
against university-wide targets; 

• identifying units and courses that are performing well, thus enabling the reward of 
excellence; 

• allowing both internal and external benchmarking (for certain course-related measures); 

• building time-series data that enables the university, the faculties and individual academics 
to see trends within their units and courses; 
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• eliminating the time-consuming manual process to get the same information (often 
repeated by different areas within the institution) to make similar decisions on units and 
courses; 

• increasing the transparency of decision-making by the university senior managers. 

Monash University has undergone an evolutionary journey from a time where decision¬ 
making by faculties and management was based on independent use of a wide range of data 
sources across the institution (using mainly Excel® spreadsheets) to the current situation where 
required information is available centrally. The implementation of the system was received 
positively across the university and this acceptance of the system is also reflected by the 
increasing demand for access to the reports. Initially, access to the system was limited to deans, 
heads of schools and departments, and faculty and departmental managers on all campuses. The 
utility and clarity of information in the reporting system has been further recognised, as there is 
now demand across the university to incorporate an increased number of metrics being reported 
in the profiles. As this is an evolutionary process, it is anticipated that the complete set of data 
sought by the faculties and managers will be incorporated in the profiles over the next two to 
three years, thus completing the university’s ambitious project for building an effective data 
management system to support evidence-based decision-making. 

The Course and Unit Profiles are viewed across the university as a positive development 
in providing an integrated access to data to aid effective management of units and courses. An 
external audit of Monash University carried out in 2006 by the Australian Universities Quality 
Agency (AUQA) also supported this view when it noted that the profiling ‘is a very positive 
initiative which is likely to improve the availability of data and the use of data to support the 
improvement of teaching’ (AUQA, 2006). 

This article has examined how a large research-intensive university in Australia has 
recently moved towards a more effective approach to evidence-based decision-making. The 
university has developed an integrated system of centrally accessible data across the whole 
institution. The system was developed to support effective decision-making at various levels 
(across the institution) related to individual units and courses, and their performance. The system 
has enabled an integration of evidence-based decision-making and quality enhancement 
processes in the institution. 
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